eXtreme Gradient Boosting (XGBoost): Better than random forest or gradient boosting

A hands-on R comparison of XGBoost, gradient boosting, random forest, lasso, and best subset regression on a slum-settlement modeling example.

• Overview
• About the Data
• 1. Extreme Gradient Boosting
• 2. Gradient boosting
• 3. Random Forest
• 4. Lasso
• 5. Best Subset
• Compare MSE

Overview

I first learned about eXtreme Gradient Boosting (XGBoost) from Professor Allan Just, then extended an earlier modeling exercise from my old blog by comparing XGBoost, Gradient Boosting (GBM), Random Forest, Lasso, and Best Subset regression.

Ensemble methods are powerful because they combine many weaker predictions into a stronger model. Random Forest averages many decorrelated decision trees built from bootstrap samples. Boosting works sequentially: each new tree focuses on the residual patterns left by the previous trees.

Correction, 2018-10-03: my first version reported a testing error almost ten times smaller than the other methods. That was a mistake. In the corrected result, XGBoost still had the lowest testing RMSE, but it was close to the other tree-based methods.

Link to the earlier version: Model Selection using Lasso and Best Subset

About the Data

In sub-Saharan Africa, where deprivations in living conditions are especially severe, slum dwellers represent an estimated 56% of the region’s urban population (UN Habitat, 2016). Measuring informal settlements reliably is a critical challenge for monitoring the Sustainable Development Goals (SDGs). The data in this example were collected by Slum Dwellers International (SDI), which was nominated for the Nobel Peace Prize in 2014.

In this exercise, we only model Share_Temporary: Share of Temporary Structure in Slums as the dependent variable. The independent variables are monitoring indicators like water, sanitation, housing conditions and overcrowding in African slum settlements. Dataset dimension is 973 x 153.

1. Extreme Gradient Boosting

• Random search: randomized parameters and update the record with best ones.
• It turns out to be a very interesting method to scan for hyperparameters. It will take a while for 100 iterations.
• The package xgboost is really fast.

library(xgboost) # Randomize and bound best_param <- list() best_seednumber <- 1234 best_rmse <- Inf best_rmse_index <- 0

set.seed(1234) # In reality, might need 100 or 200 iterations for (iter in 1:10) { param <- list(objective = "reg:squarederror", # For regression eval_metric = "rmse", # rmse is used for regression max_depth = sample(6:10, 1), eta = runif(1, .01, .1), # Learning rate, default: 0.3 subsample = runif(1, .6, .9), colsample_bytree = runif(1, .5, .8), min_child_weight = sample(5:10, 1), # These two are important max_delta_step = sample(5:10, 1) # Can help to focus error # into a small range. ) cv.nround <- 1000 cv.nfold <- 5 # 5-fold cross-validation seed.number <- sample.int(10000, 1) # set seed for the cv set.seed(seed.number) mdcv <- xgb.cv(data = dtrain, params = param,

nfold = cv.nfold, nrounds = cv.nround, verbose = F, early_stopping_rounds = 8, maximize = FALSE)
min_rmse_index <- mdcv\(best_iteration
  min_rmse &lt;-  mdcv\)evaluation_log[min_rmse_index]$test_rmse_mean
if (min_rmse < best_rmse) { best_rmse <- min_rmse best_rmse_index <- min_rmse_index best_seednumber <- seed.number best_param <- param } }

• The best tuning parameters
  

##          objective eval_metric max_depth     eta subsample colsample_bytree
## 1 reg:squarederror        rmse         9 0.09822      0.64           0.6853
##   min_child_weight max_delta_step best_rmse_index best_rmse best_seednumber
## 1                6              8              56    0.2102            3660

• MSE
  

## [1] 0.04237

• Feature Importance
  

importance_matrix <- xgb.importance(feature_names = colnames(X_train), 
                                    model = xg_mod)
# Use `xgb.plot.importance`, which create a _barplot_ or use `xgb.ggplot.importance`
library(Ckmeans.1d.dp) # for xgb.ggplot.importance
xgb.ggplot.importance(importance_matrix, top_n = 15, measure = "Gain")

• Plot only 2 trees as an example (use trees= 1)
  

library("DiagrammeR")
xgb.plot.tree(model = xg_mod, trees = 1, feature_names = colnames(X_train))

• Plot all trees on one tree and plot it: A huge plot

xgb.plot.multi.trees(model = xg_mod, n_first_tree = 1, feature_names = colnames(X_train))